Model comparison methods.

The question of how one should choose among competing explanations (models) of observed data is at the core of science. Model comparison is ubiquitous and arises, for example, when a toxicologist must decide between two dose-response models or when a biochemist needs to determine which of a set of enzyme-kinetics models best accounts for observed data. Over the decades, a number of criteria that are thought to be important for model comparison have been proposed (e.g., Jacobs & Grainger, 1994). They include (a) falsifiability (Popper, 1959): whether there exist potential observations that are incompatible with the model; (b) explanatory adequacy: whether the theoretical account of the model helps to make sense of observed data but also established findings; c) interpretability: whether the components of the model, especially its parameters, are understandable and are linked to known processes; (d) faithfulness: whether the model’s ability to capture the underlying regularities comes from the theoretical principles the model purports to implement, not from the incidental choices made in its computational instantiation; (e) goodness of fit: whether the model fits the observed data sufficiently well; (f) complexity or simplicity; whether the model’s description of observed data is achieved in the simplest possible manner; and (g) generalizability: whether the model provides a good prediction of future observations. Although each one of these seven criteria is important in its own way, modern statistical approaches to model comparison consider only the last three (goodness of fit, complexity, generalizability), largely because they lend themselves to quantification. The other four criteria have yet to be formalized and it is not clear how some even could be or should be (e.g., interpretability). The purpose of this chapter is to provide a tutorial on state-of-the-art statistical model comparison methods. We walk the reader through the reasoning underlying their development so that how and why a method performs as it does can be understood. The chapter is written for researchers who are interested in computational modeling but are primarily involved in empirical work. We begin by discussing the statistical foundations of model comparison.